Automated surgical and interventional procedures

ABSTRACT

Described herein are an apparatus and methods for automating subtasks in surgery and interventional medical procedures. The apparatus consists of a robotic positioning platform, an operating system with automation programs, and end-effector tools to carry out a task under supervised autonomy. The operating system executes an automation program, based on one or a fusion of two or more imaging modalities, guides real-time tracking of mobile and deformable targets in unstructured environment while the end-effector tools execute surgical interventional subtasks that require precision, accuracy, maneuverability and repetition. The apparatus and methods make these medical procedures more efficient and effective allowing a wider access and more standardized outcomes and improved safety.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a divisional application of, and claims the benefitof priority under 35 U.S.C. § 120 from, U.S. application Ser. No.13/931,371, filed Jun. 28, 2013, since issued on Dec. 29, 2015 as U.S.Pat. No. 9,220,570, which claims the benefit of priority under 35 U.S.C.§ 119(e) from U.S. Ser. No. 61/666,399, filed Jun. 29, 2012, the entirecontents of which are incorporated herein by reference.

BACKGROUND OF THE INVENTION

Field of the Invention

This invention is related to the field of robotic surgery, namely, fullor partial automation of surgical tasks.

Description of the Related Art

What is available in the market as so called “robotic surgery” istypically robot-assisted surgery because the surgeon and the robotinteract through a master-slave paradigm. The slave robot is constrainedto follow direct commands from the surgeon master with little autonomy.While this method is reliable, it constrains the speed and dexterity ofthe slave robot to that of the surgeon. None of the currently availableinvasive surgical systems utilize true automation during the procedure.Moreover, no previous approach combines both visible light images withother modalities to control the robot.

SUMMARY OF THE INVENTION

The currently available master-slave mode of operation limits the robotto the operating surgeon's dexterity and skill, which may be inefficientfor certain subtasks that require high precision, dexterity, andrepetition (e.g. suturing) when compared to autonomous control.Moreover, supervised automation improves on limitations of eachindividual surgeon's experience-based adaptive and visual processingability with an evidence-based decision support algorithm built onmaster surgeon's ability and sub-surface tissue information from asecondary imaging source.

Surgical subroutines that require high dexterity, precision, andrepetition will be identified and chosen for automation. Clinical data(visual recording, robot movement, patient outcome, etc.) from bothmanual and robot-assisted performances of these routines by expertsurgeons will be analyzed to identify the automation-criticalinformation (e.g. key reference points, visual references, organ/vessellocation/movement, etc.) and the optimal movement pattern forautomation. The automation program will be designed so that theautomation program can adapt the surgical task to different patientsgiven the aforementioned critical information. The automated routinewill not only mimic the clinical data, but also learn from expertsurgeons' performances and improve upon the surgeons' performance totake advantage of the efficiency and effectiveness of the robot.

The automation program may make use of visual servoing with real-timevisual feedback from one or more cameras, either endoscopic orexternally mounted, which are capable of providing visible and/ornon-visible spectrum images. An example of non-visible spectrum image isthe near-infrared fluorescent (NIR) image. 3D information may beprovided along with visual information through a dedicated sensor orextraction from visual information. For example, if two cameras areused, 3D depth information could be extracted from stereo triangulationalgorithm. 3D depth information could also be obtained throughstructured-light 3D scanners, or through light-field cameras.

Due to the often dynamic and unstructured nature of the surgicalenvironment, optical data alone may not be sufficient for robustreal-time, high fidelity tracking of mobile and deformable targets;therefore, the visual data may be augmented, fused or accompanied byother sensors as needed (e.g. infrared-camera, haptic sensor, etc.). Thevisual image, along with other sensory and critical data, will be fedinto the automation program's control system that will move the robotand the tools to perform the desired surgical task. More than oneautomation program may be generated for each surgical procedure, witheach program accepting different combination of sensors and criticaldata to accommodate different surgery conditions, but accomplishing thesame surgical goal. A single program may be made to accept multiplecombinations of sensors and data as well.

During surgery, the robot will be able to operate under one of threemodes of operations: master-slave, semi-autonomous, and supervisedautonomous. In master-slave mode, the surgeon directly controls therobot's motions to perform tasks that cannot be done with the other twomodes such as preparing the surgical scene for an autonomous program(e.g. placing tissue markers). In the semi-autonomous mode, the surgeonwill provide the robot with action commands (e.g. place suture inspecific location, cut tissue in a line, tie a knot) that the robot willperform using autonomously calculated trajectories and tool actuations.That is, after preparation of surgical site, the surgeon is stillinvolved in decision making and command specification interactively.This interaction may be implemented through a graphical user interface,where the surgeon outlines suture locations such that the program canvisually track target locations and generate robot trajectories to reachthe targets. The part where the surgeon interacts with the programdefines semi-autonomy. In the supervised autonomous mode, the surgeononly provides the robot with an overall goal (e.g. perform anastomosis)and the autonomous program determines the actions necessary to completethe goal (e.g. location, number, tension, and order of sutures to place)without any input from the surgeon. That is, after preparation of thesurgical site, the program picks the target location and proceedsautomatically. The surgeon's role is primarily safety supervision.Throughout the surgery, the surgeon may employ any of these three modesas appropriate, and at any time in the operation, the surgeon mayinterrupt the robot's motion and take master-slave control of the robot.

An example of one surgical subtask that embodiments of this inventioncould significantly benefit is anastomosis. Anastomosis isconventionally performed manually or more recently using robots throughmaster-slave control, but both techniques are time consuming andcumbersome due to the high amount of dexterity, precision, andrepetition required for the procedure. There is great potentialimprovement to be had from automating this task because of thesecharacteristics.

The present technology has the potential to improve upon other surgicalprocedures requiring precision, repetition, maneuverability, andreproducibility, including but not limited to placement(screwing/fixation) of bone implants, tissue dissections, biopsies,vitreo-retinal surgeries, microsurgical and/or vascular anastomosis,brachytherapy, and skin closure.

Embodiments disclosed herein provide for a system for performing anautomated surgical procedure. The system includes a sensor that providesinformation regarding a surgical field, a user interface configured toreceive commands issued by a surgeon, a feedback device configured torelay information to the surgeon, a surgical tool having an end portionused for performing a surgical task, a surgical robot that is coupled tothe surgical tool and that positions and orients the surgical tool, atrack processing module implemented by processing hardware andconfigured to receive sensor data from the sensor, identify positions inat least one of a target tissue, surrounding tissues and the surgicaltool end portion based on the sensor data, track the identifiedpositions in at least one of the target tissue, the surrounding tissuesand the tool end effector, and a control module implemented by theprocessing hardware and configured to process data received from thesensor, the track processing module, and the user interface via anautomation program, to generate and send commands to the surgical robot.

According to another embodiment of the system, the system furthercomprises a plurality of sensors that provide information regarding thesurgical field.

According to another embodiment of the system, the sensor is one of acamera, a near-infrared fluorescent (NIR) camera, a depth camera, astructured light 3D scanner.

According to another embodiment of the system, the feedback device is adisplay configured to show visual cues or images or an auditory device.

According to another embodiment of the system, the track processingmodule is further configured to track the identified positions in atleast one of the target tissue, the surrounding tissues and the tool endeffector, using near-infrared fluorescent (NIR) markers.

According to another embodiment of the system, the surgical robot thatis detachably coupled to the surgical tool.

According to another embodiment of the system, the surgical robot iscoupled to a movement mechanism that moves the surgical robot in and outof the surgical field.

According to another embodiment of the system, the automation program issemi-autonomous.

According to another embodiment of the system, the automation program issupervised autonomous.

According to another embodiment of the system, the control module isconfigured to disable the automation program and implement amaster-slave mode.

According to another embodiment of the system, the control module isconfigured to interrupt the automation program based on surgeon input.

According to another embodiment of the system, the control module isfurther configured to implement visual servoing correction.

According to another embodiment of the system, the automation program isconfigured to implement anastomosis.

According to another embodiment of the system, the control module isfurther configured to further generate the commands based on at leastone of a no-fly zone, a remote center of motion, and a velocity/forcelimit.

According to another embodiment of the system, the automation program isconfigured to join tissue by generating and sending commands to thesurgical robot.

According to another embodiment of the system, the joining of the tissueis performed via suture, clips, staples or adhesive.

According to another embodiment of the system, the control module isfurther configured to implement visual servoing correction to bring thetool end portion to a target.

According to another embodiment of the system, the positions identifiedby the track processing module are three-dimensional positions.

Embodiments disclosed herein further provide for a computer implementedmethod of generating an automated surgical program. The method includesthe steps of processing clinical data to produce a 3D spatial andtemporal data of a surgery, obtaining surgical robot specifications andclinical parameters, generating the automated surgical program based onthe 3D spatial and temporal data, the surgical robot specifications, andthe clinical parameters.

According to another embodiment of the method, the clinical dataincludes at least one of visual data obtained from a camera orendoscope, kinematic data, or haptic data.

According to another embodiment of the method, the clinical dataincludes at least one of patient condition, vitals, and outcome of thesurgery.

According to another embodiment of the method, the clinical dataincludes surgeon experience.

According to another embodiment of the method, the processing of theclinical data produces the 3D spatial and temporal data of the surgerybased on considering correlations between tool motions and surgicaloutcomes.

According to another embodiment of the method, the processing of theclinical data produces the 3D spatial and temporal data of the surgerybased on considering differences between surgeon experience levels toproduce more effective movements

According to another embodiment of the method, the 3D spatial andtemporal data includes at least one of tool motion, tool positioning,location and movement of vital organs or structures, viable referencepoints, and tissue deformation.

According to another embodiment of the method, the surgical robotspecifications include velocity and precision of the surgical robot.

According to another embodiment of the method, the generated automatedsurgical program includes surgical motions absent in clinical data butimplementable by the surgical robot.

According to another embodiment of the method, the clinical parametersincludes tissue characteristics.

According to another embodiment of the method, the tissuecharacteristics include expected movement and rigidity of the tissue orabsorbability of dyes of the tissue.

According to another embodiment of the method, the automated surgicalprogram includes: preferred movement patterns, critical operationinformation, a control module configured to instructs robot motionsbased on a combination of sensor information and the critical operationinformation.

According to another embodiment of the method, the critical operationinformation includes at least one of vital organ positions, referencepoints or markers, sensor data, and surgeon input.

According to another embodiment of the method, the control module isfurther configured to further generate the commands based on motionconstraints that include at least one of remote center of motion, no-flyzones, and velocity limits.

According to another embodiment of the method, the control module isfurther configured to further generate the commands based on tissuedeformation or movement models.

According to another embodiment of the method, wherein the automatedsurgical program is updatable with additional clinical data.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows the overall workflow of utilizing the invention in roboticsurgery;

FIG. 2 shows the overall structure of the embodiment of the invention insemi-autonomous mode where the surgical tasks are partially automated;

FIG. 3 shows the embodiment of the system in the master-slaverobot-assisted mode;

FIG. 4 shows an embodiment of the system with supervised autonomy;

FIGS. 5A and 5B show example infrared images with fluorescent markers,where FIG. 5A illustrates an infrared image with fluorescent markers andFIG. 5B illustrates a binary image with intensity threshold;

FIGS. 6A and 6B show example current and target images for use in visualservoing effecting image-coordinate error correction, where FIG. 6Aillustrates the current image and FIG. 6B illustrates the target image;

FIG. 7 shows an embodiment utilizing dual-mode endoscope for automationof anastomosis;

FIG. 8 shows how the tissue and tool may be marked with fluorescentmarkers with a view of an organ and an anastomosis tool for anastomosiswith NIR and a biodegradable clip;

FIG. 9 shows how a specialized tool for automated anastomosis may workwith a view of an anastomosis tool attaching a clip to the organ;

FIGS. 10A, 10B and 10C show the two images the dual-mode endoscope mayreceive, and how they may be overlaid, where FIG. 10A illustrates avisible spectrum image, FIG. 10B illustrates a NIR spectrum image, andFIG. 10C illustrates a visible image and NIR image overlaid;

FIGS. 11A and 11B show an embodiment of a special clip made foranastomosis, where FIG. 11A illustrates the biodegradable clip, theclasp, and the tissue and FIG. 11B illustrates how the clip pierces bothtissues, and the clasp is tightened on one end of the clip to hold thetissues together;

FIG. 12 shows the overall procedure for developing the automatedsurgical program from clinical data; and

FIG. 13 illustrates a block diagram of a computing device according toone embodiment.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The embodiments of the invention describe a system for automation ofsurgical tasks. The embodiments consist of generating an automatedsurgical program based on clinical data, then applying the automatedsurgical program to patients during surgery.

FIG. 1 shows the overall workflow of utilizing the proposed system in asurgical operation. The surgeon starts 100 the surgery by utilizingmanual tools 101, either performing segments of the operation that canbe done efficiently manually, or preparing the surgical site for therobot 102. Once the patient has been prepared, the surgeon thenintroduces the robot 103 into the patient, and begins the robotic modeof operation 104. When deployed, the robot may be set to work with oneof three modes of operation: master-slave 105, where the surgeoncontrols the robot's motion directly through controllers;semi-autonomous 106, where the robot performs tasks under the directionof a surgeon; or supervised autonomous 107, where the robot performs atask autonomously under the supervision of the surgeon. Beforeactivating the semi-autonomous or supervised autonomous mode ofoperation, the surgeon would prepare the surgical site and the surgicalsystem as required 108. The surgeon may also freely switch between thedifferent modes of operation 109, until the robot is no longer needed ormust be removed to continue the surgery 111. After the robot's removal,the surgeon may either continue the surgery using manual tools 101,bringing the robot back if needed 103, or bring the operation to an endby performing any final tasks 112.

FIGS. 2, 3 and 4 represent the different modes of the operation for theproposed system. In the semi-autonomous mode (FIG. 2) the surgeon 200provides commands to the automation program 208 during the operation.The automation program 208 receives the tracking information from theimage-based tracking module 207, combines the tracking information withthe intraoperative commands from the surgeon 200 and thepatient-specific parameters 210 and sends appropriate commands to therobot in real-time in order to control the surgical robot 201 and thesurgical tool(s) 211 (which may or may not be specialized for thecurrent procedure) to obtain a predetermined goal (e.g. anastomosis).The surgeon 200 can be given visual, audio or haptic feedback 212 whilehe/she is looking at the visual display 209, and interacts with thesurgical robot as a supervisor 213, taking over control through a masterconsole whenever required.

In master-slave mode (FIG. 3), the surgeon 300 controls the surgicaltool through master-slave control 314 of a robot 301. The surgeonreceives visual feedback through the visual display 309 and may also beprovided with other visual, audio or haptic feedback 312 but the controlloop is solely closed through the surgeon.

In supervised autonomous mode (FIG. 4), the control loop is solelyclosed via autonomous program 408 that utilizes image-based tracking andpatient-specific parameters 410 except when the surgeon 400 stops theautonomous control and takes over control 413 to prevent a complication,correct for a wrong action, or other reasons.

In surgery, the surgeon must prepare the surgical scene using manualtools or the robot to enable the automation program to take over. Thismay include but is not limited to: placing the tools in the proximity ofthe target organ 202, moving the camera 203 to provide vision of thetarget organ 202, marking key reference points, setting up extra sensormodules 205, marking the work area, and marking the vitaltissues/organs. Once the surgical scene is set up, the semi-autonomousand supervised autonomous modes of operation may be used as appropriate.

A visible light source 204 lights the surgical scene, allowing thecamera 203 to record live images of the procedure. The image acquisitionand control module 206 captures and digitizes the images from theendoscope/camera 203 and provides them to the image-based trackingmodule 207 and the visual display 209. The visual display 209 providesan image feed of the acquired visual images; the visual display 209 canalso display an augmented reality image by overlaying the video withinformation from the extra sensors 205 or from the image-based trackingmodule 207. The image-based tracking module 207 applies image processingalgorithms to track the tools and reference points. These tasks would beperformed by a computer that is connected to the sensors and containsthe software for image acquisition 206, automation program 208,image-based tracking module 207, and processing feedback 212.

The extra sensor modules 205, which are used as needed to make theautomation program more robust, can send information from the extrasensor modules 205 to either the image acquisition module 206 ordirectly to the automation program 208, depending on the nature of thesensor. The extra sensor modules may also send information from theextra sensor modules 205 to the visual display 209 for overlaying withvideo or be sent to the surgeon console to provide visual, audio, orhaptic feedback 212.

In one embodiment of the invention, the surgeon selects a series ofautomation programs from a library of available automation programs. Anexample of an automation program is one that performs a suturing taskwhere one or more points on different tissues must be sutured orstitched together, that is, the surgical tool must be positioned withrespect to the tissue to perform suturing.

In one embodiment of the invention, the automation program utilizes animage-based visual servoing system, where the robotic tool is controlledin closed-loop using an image-based control law. In visual servoing, thedifference between the desired image, which depicts the tool at thetarget location, and the current image, which depicts the current tooland the target location, is used to compute the error in imagecoordinates. This error in image coordinates is used to generate themotion of the robotic tool towards the target position. As the robotictool gets closer to the target location in the surgical field, the errorin the image space gets smaller. At the final control loop iteration,the error approaches zero, at which point the tool has reached thetarget location in both the image coordinates and the Cartesian robotcoordinates. This is the core of the image-based visual servoing controlloop. If stereo camera system is used, the coordinates of the left andright images could be augmented to control more degrees of freedom (DOF)of the robotic tool.

One embodiment of the invention uses images that contain the visiblespectrum of the surgical field and/or other non-visible light contentsuch as near-infrared spectrum (NIR, 700˜1100 nm). For example, beforethe autonomous program is activated, the surgeon may place NIR markersat target locations that will be tracked using an NIR camera. Thedistinguishability of the NIR markers from the visual spectrum images,along with the ability of near-infrared spectrum to pass through bloodand tissue, allows for more robust real-time tracking of target tissuesin the dynamic surgical environment (e.g. deforming soft tissue).Multi-spectral optical imaging may also be used to detect sub-surfacetissue information that assist in optimal targeting.

Automation of anastomosis is described in an embodiment of thisinvention, where tubular organs such as the intestine are joined usingsutures, clips, glue, or staples. In semi-autonomous anastomosis usingsutures, the surgeon first prepares the tubular organs in pre-definedorientations and marks suture locations or circumference of tubes forthe program to visually track. The automation program then autonomouslymoves the robotic tool to the selected suture locations and performssuturing. In autonomous anastomosis using sutures, the program firstdetects the optimal suture locations based on properties of the tissue(e.g. mechanical, optical, geometric) and kinematic and dynamiccharacteristics of the robotic tool for optimal dexterity. Once theoptimal suture locations are detected, the autonomous program brings thetool to the suture location and performs suturing.

FIGS. 5 and 6 show one embodiment of this invention that utilizes visualservoing and NIR markers to perform anastomosis. In this embodiment, thesurgeon places fluorescent NIR markers 500 at the target suturelocations to prepare the surgical site for the autonomous program. Thevisual system obtains both visible spectrum and near-infrared spectrumimages (FIG. 5A), allowing the visual servo to reliably track theNIR-marked tool 501 and suture locations 502 in real-time through theNIR markers 500. One example of image processing that may be performedto aid in tracking is an infrared-threshold binary image (FIG. 5B),which clearly differentiates the marked areas from the non-marked areas.The visual servo then moves 603 the robotic tool 602 towards the suturesite 601 (FIG. 6A) so that the error in the image and Cartesiancoordinate space approaches zero (FIG. 6B). Once a suture site isreached, the autonomous program places a suture before moving onto thenext suture site.

To further aid in tracking of tissues in the dynamic and deformingsurgical environment, certain embodiments of this invention may havemeans of obtaining 3D information about the surgical workspace. Oneembodiment of this means uses two cameras, which allows for theextraction of 3D depth information through a stereo triangulationalgorithm. Another embodiment involves using structured-light 3Dscanners to obtain 3D information. Another embodiment involves obtaining3D information through light-field cameras.

FIG. 7 shows the system diagram of the embodiment that utilizes visualservoing with NIR markers for anastomosis. Fluorescent markers aredeployed on the organ 702 (e.g. two sides of a bile duct to beanastomosed) in manual mode and two light sources 704 and 715 illuminatethe scene. One light source 704 is a visual light source that makes itpossible to acquire normal images of the organs. The other light source715 is a narrow-band source of light (e.g. in the near infrared range)that is chosen according to the excitation wavelength of the fluorescentmaterial. Both visible light and fluorescent light images are capturedby the dual-mode endoscope 703 and sent to the image acquisition andcontrol module 706, which will then send the images to the visualdisplay 709 for overlaying and to the image-based tracking module 707for processing. The automation program's visual servoing control system708 utilizes the fluorescent markings to become more robust, allowingthe automation program to move the robot 701 and the specializedanastomosis tool 711 appropriately to carry out the desired procedure(anastomosis).

FIGS. 8, 9, and 10 show an embodiment of the invention that performsanastomosis with NIR markers and biodegradable clips. To prepare for thesupervised autonomous mode of operation, fluorescent markers 816 aredelivered around the anastomosis site 802, and optionally, the tool 811.The tool deploys biodegradable clips 817 (more detail is provided inFIG. 11) that can be used to perform the anastomosis. In FIG. 9, theautonomous program is provided with images of the fluorescent markers916 that, along with other sensor data, are used to guide the tool 911to the anastomosis site 902, where clips 917 will be deployed to performthe anastomosis. FIG. 10 demonstrates the potential benefit of usingfluorescent markers in the presence of visual obstructions 1018 in thesurgical field. While the obstruction would impair vision of theanastomosis site in the visible spectrum (FIG. 10A), certain fluorescentdyes emit infrared light that can pass through obstructions (FIG. 10B).By combining information from different spectrum of light (FIG. 10C),the visual tracking system is made more robust.

FIG. 11 shows one embodiment of a biodegradable clip 1117 used toperform the anastomosis. The clip pierces through the two tissues 1102to be joined, and is fixed in place by tightening a biodegradable clasp1119 around the tail of the biodegradable clip 1117.

FIG. 12 represents the general workflow for developing the automatedsurgical program from clinical data. Clinical data 120 is processed toobtain a set of raw data 121. This consists of visual data from camerasor endoscopes, kinematic and haptic information if the surgery isperformed robotically, and other relevant data, such as the patient'scondition throughout the surgery, outcome, vitals, etc. These data areanalyzed, either manually or by using computer algorithms such aspattern recognition, to produce a set of spatiotemporal information 122about the surgery. This set contains tool motion, tool positioning,location and movement of vital organs/structures, viable referencepoints, tissue deformation, and other information, such as correlationbetween certain motions and patient outcomes. Inefficient movements canalso be identified at this stage by comparing the movements of surgeonsof varying experience, which can be removed during programming andidentified for training surgeons in the future. This may be realized bytechniques from robotic imitation-learning, where sensory data fromexpert operators are gathered while performing similar maneuvers. Thesensory data, such as trajectories of the input device, are first scaledand normalized, then parameterized. The parameters are learned, e.g.,using linear subspace methods such as Principle Component Analysis (PCA)from maneuver repetitions of the same task. Each expert maneuver canthen be represented by linear combinations of different parametriccurves. The movements may be further optimized by incorporatingmovements that surgeons would normally not make due to dexterityconstraints of their hands. This spatiotemporal data of the procedure isthen combined with the surgical robot's capability 123 (e.g. speed anddexterity) and tissue characteristics 124 (e.g. expected movement,tissue rigidity) to produce the automation program 125 specific to asurgical procedure. The program consists of the movement patterns in theprocedure, a control system that combines different sensory informationto produce the movement patterns, a set of critical information (e.g.reference points, vital organs/vessels) that must be provided, a set ofconstraints, such as speed limits and spatial constraints, anddeformation/movement models of the tissues involved. An updating methodmay also be implemented to incorporate more expert surgeons' clinicaldata to help improve this automated procedure over time. Each of theprogram or algorithm based elements of the above noted description canbe implemented by hardware such as the hardware found in the descriptionof FIG. 13. In FIG. 12, the computer 1299 includes a CPU 1200 whichperforms the processes described above. The process data andinstructions may be stored in memory 1202. These processes andinstructions may also be stored on a storage medium disk 1204 such as ahard drive (HDD) or portable storage medium or may be stored remotely.Further, the claimed advancements are not limited by the form of thecomputer-readable media on which the instructions of the inventiveprocess are stored. For example, the instructions may be stored on CDs,DVDs, in FLASH memory, RAM, ROM, PROM, EPROM, EEPROM, hard disk or anyother information processing device with which the system communicates,such as a server or computer.

Further, the claimed advancements may be provided as a utilityapplication, background daemon, or component of an operating system, orcombination thereof, executing in conjunction with CPU 1200 and anoperating system such as Microsoft Windows 7, UNIX, Solaris, LINUX,Apple MAC-OS and other systems known to those skilled in the art.

CPU 1200 may be a Xenon or Core processor from Intel of America or anOpteron processor from AMD of America, or may be other processor typesthat would be recognized by one of ordinary skill in the art.Alternatively, the CPU 1200 may be implemented on an FPGA, ASIC, PLD orusing discrete logic circuits, as one of ordinary skill in the art wouldrecognize. Further, CPU 1200 may be implemented as multiple processorscooperatively working in parallel to perform the instructions of theinventive processes described above.

The computer 1299 in FIG. 13 also includes a network controller 1206,such as an Intel Ethernet PRO network interface card from IntelCorporation of America, for interfacing with network 1250. As can beappreciated, the network 1250 can be a public network, such as theInternet, or a private network such as an LAN or WAN network, or anycombination thereof and can also include PSTN or ISDN sub-networks. Thenetwork 1250 can also be wired, such as an Ethernet network, or can bewireless such as a cellular network including EDGE, 3G and 4G wirelesscellular systems. The wireless network can also be WiFi, Bluetooth, orany other wireless form of communication that is known.

The computer 1299 further includes a display controller 1208, such as aNVIDIA GeForce GTX or Quadro graphics adaptor from NVIDIA Corporation ofAmerica for interfacing with display 1210, such as a Hewlett PackardHPL2445w LCD monitor. A general purpose I/O interface 1212 interfaceswith a keyboard and/or mouse 1214 as well as a touch screen panel 1216on or separate from display 1210. General purpose I/O interface alsoconnects to a variety of peripherals 1218 including printers andscanners, such as an OfficeJet or DeskJet from Hewlett Packard. Theperipheral elements previously described in the above exemplaryembodiments may be embodied by the peripherals 1218 in the exemplaryembodiment of FIG. 13.

A sound controller 1220 may also be provided in the computer 1299, suchas Sound Blaster X-Fi Titanium from Creative, to interface withspeakers/microphone 1222 thereby providing sounds and/or music. Thespeakers/microphone 1222 can also be used to accept dictated words ascommands for controlling the robot-guided medical procedure system orfor providing location and/or property information with respect to thetarget property.

The general purpose storage controller 1224 connects the storage mediumdisk 1204 with communication bus 1226, which may be an ISA, EISA, VESA,PCI, or similar, for interconnecting all of the components of therobot-guided medical procedure system. A description of the generalfeatures and functionality of the display 1210, keyboard and/or mouse1214, as well as the display controller 1208, storage controller 1224,network controller 1206, sound controller 1220, and general purpose I/Ointerface 1212 is omitted herein for brevity as these features areknown.

Obviously, numerous modifications and variations of the presentdisclosure are possible in light of the above teachings. It is thereforeto be understood that within the scope of the appended claims, theinvention may be practiced otherwise than as specifically describedherein. For example, advantageous results may be achieved if the stepsof the disclosed techniques were performed in a different sequence, ifcomponents in the disclosed systems were combined in a different manner,or if the components were replaced or supplemented by other components.The functions, processes and algorithms described herein may beperformed in hardware or software executed by hardware, includingcomputer processors and/or programmable processing circuits configuredto execute program code and/or computer instructions to execute thefunctions, processes and algorithms described herein. A processingcircuit includes a programmed processor, as a processor includescircuitry. A processing circuit also includes devices such as anapplication specific integrated circuit (ASIC) and conventional circuitcomponents arranged to perform the recited functions.

The functions and features described herein may also be executed byvarious distributed components of a system. For example, one or moreprocessors may execute these system functions, wherein the processorsare distributed across multiple components communicating in a network.The distributed components may include one or more client and/or servermachines, in addition to various human interface and/or communicationdevices (e.g., display monitors, smart phones, tablets, personal digitalassistants (PDAs)). The network may be a private network, such as a LANor WAN, or may be a public network, such as the Internet. Input to thesystem may be received via direct user input and/or received remotelyeither in real-time or as a batch process. Additionally, someimplementations may be performed on modules or hardware not identical tothose described. Accordingly, other implementations are within the scopethat may be claimed.

It should be noted that, as used in the specification and the appendedclaims, the singular forms “a,” “an,” and “the” include plural referentsunless the context clearly dictates otherwise.

What is claimed is:
 1. A computer implemented method of generating,using processing circuitry and memory of the computer, operationinstructions for an autonomous surgical procedure for a particularsurgical robot for a particular surgery, and of performing theautonomous surgical procedure, the method comprising: processing, usingthe processing circuitry, clinical data from a plurality of previouslyperformed surgeries to generate 3D spatial and temporal data defining amodel surgery process; obtaining, using the processing circuitry andfrom the memory, surgical robot specifications of the particularsurgical robot, the surgical robot specifications defining capabilitiesof the particular surgical robot; obtaining, using the processingcircuitry and from the memory, clinical parameters of the particularsurgery, the clinical parameters defining characteristics of a portionof a body on which the particular surgery is performed; generating,using the processing circuitry, the operation instructions for theautonomous surgical procedure based on the 3D spatial and temporal data,the surgical robot specifications, and the clinical parameters; andperforming the autonomous surgical procedure using the generatedoperation instructions, without additional operation instructions from asurgeon.
 2. The method of claim 1, wherein the clinical data includes atleast one of visual data of the previously performed surgeries obtainedfrom a camera or endoscope, kinematic data, or haptic data, or theclinical data includes at least one of patient condition, vitals, andoutcome of the previously performed surgeries.
 3. The method of claim 1,wherein the clinical data includes data associated with surgeonexperience.
 4. The method of claim 1, wherein the processing of theclinical data generates the 3D spatial and temporal data defining themodel surgery process based on considering at least correlations betweentool motions and surgical outcomes.
 5. The method of claim 3, whereinthe clinical data from a plurality of previously performed surgeriesincludes at least one of tool motion, tool positioning, location andmovement of vital organs or structures, viable reference points, andtissue deformation.
 6. The method of claim 1, wherein the surgical robotspecifications of the particular surgical robot include velocity of theparticular surgical robot.
 7. The method of claim 1, wherein thegenerated operation instructions for the autonomous surgical procedureinclude surgical motions that are absent in clinical data from theplurality of previously performed surgeries but are neverthelessimplementable by the particular surgical robot.
 8. The method of claim1, wherein the clinical parameters of the particular surgery includetissue characteristics of the portion of the body on which theparticular surgery is performed.
 9. The method of claim 8, wherein thetissue characteristics of the portion of the body on which theparticular surgery is performed include expected amount of movement andrigidity of tissue of the portion or absorbability of dyes in the tissueof the portion.
 10. The method of claim 1, further comprising:instructing, using processing circuitry of the particular surgicalrobot, robot motions based on sensor information and the operationinstructions.
 11. The method of claim 10, wherein the sensor informationincludes at least one of vital organ positions, reference points ormarkers, and sensor data.
 12. The method of claim 10, further comprisinggenerating commands based on motion constraints that include at leastone of remote center of motion, no-fly zones, and velocity limits. 13.The method of claim 10, further comprising generating commands based ontissue deformation or tissue movement models.
 14. The method of claim 1,further comprising: updating the generated operation instructions forthe autonomous surgical procedure with additional clinical data.
 15. Themethod of claim 1, wherein the clinical data includes at least one ofvisual data of the previously performed surgeries obtained from a cameraor endoscope, kinematic data, or haptic data, and the clinical dataincludes patient condition throughout a surgery of the previouslyperformed surgeries.
 16. A computer implemented method of generating,using processing circuitry and memory of the computer, operationinstructions for an autonomous surgical procedure for a particularsurgical robot for a particular surgery, and of performing theautonomous surgical procedure, the method comprising: processing, usingthe processing circuitry, clinical data from a plurality of previouslyperformed surgeries to generate 3D spatial and temporal data defining amodel surgery process; obtaining, using the processing circuitry andfrom the memory, surgical robot specifications of the particularsurgical robot, the surgical robot specifications defining capabilitiesof the particular surgical robot, or obtaining, using the processingcircuitry and from the memory, clinical parameters of the particularsurgery, the clinical parameters defining characteristics of a portionof a body on which the particular surgery is performed; generating,using the processing circuitry, the operation instructions for theautonomous surgical procedure based on the 3D spatial and temporal data,the surgical robot specifications, and the clinical parameters; andperforming the autonomous surgical procedure using the generatedoperation instructions, without additional operation instructions from asurgeon.